The State of the Art: Object Retrieval in Paintings using Discriminative Regions

The objective of this work is to recognize object categories (such as animals and vehicles) in paintings, whilst learning these categories from natural images. This is a challenging problem given the substantial differences between paintings and natural images, and variations in depiction of objects in paintings [5] – see figure 1. Contributions. (i) We show that object category classifiers learnt using Fisher Vectors [4] extracted from natural images can retrieve paintings containing that category with some success; (ii) we then introduce a method of re-ranking these retrieved paintings based on spatial consistency of Mid-Level Discriminative Patch (MLDP) correspondences with the original training images and show that the precision of the top ranked paintings (i.e. the ones that would appear on the first webpage in an image search) can be significantly improved using this method. Motivation. Obtaining paintings with a particular object is of interest to Art Historians who currently find paintings manually or from memory. They can then study the change in the depiction style over time or determine when an object first appeared in paintings. Summary of method. Object category classifiers are learnt from training sets of natural images (e.g. PASCAL VOC) and applied to paintings. The top ranked paintings for each category are re-ranked based on their spatial consistency with the natural images as follows: (i) discriminative regions are extracted from the natural images using the method of Aubry et al. [2] (figure 2); (ii) these regions are used to learn LDA [3] classifiers which are applied as sliding window detectors to the top ranked paintings to find matching regions and a RANSAC style algorithm is used to remove outlying matches (figure 3); (iii) each painting is scored by the maximum number of inlying matches shared with a natural image and are re-ranked accordingly (figure 4).